Skip to content

EmergencyReparentShard: filter lagging candidates from relay-log wait, tolerate partial failures#18707

Draft
timvaillancourt wants to merge 32 commits into
vitessio:mainfrom
timvaillancourt:ers-handle-minority-lagging
Draft

EmergencyReparentShard: filter lagging candidates from relay-log wait, tolerate partial failures#18707
timvaillancourt wants to merge 32 commits into
vitessio:mainfrom
timvaillancourt:ers-handle-minority-lagging

Conversation

@timvaillancourt

@timvaillancourt timvaillancourt commented Sep 30, 2025

Copy link
Copy Markdown
Contributor

Description

When running Vitess in production, one of the most common problems preventing EmergencyReparentShard operations from occurring quickly and without error is outlier lagging tablets. Some situations where this occurs:

  1. Tablets coming out of restore - often lagging
  2. REPLICAs that struggle to keep up in replication - typically due to query load
  3. Background tablets doing expensive jobs, such as RDONLYs

Today, tablets lagging in these scenarios are likely to cause ERS to take longer and/or timeout 😱

After StopReplicationAndGetStatus completes during ERS, replication is stopped on all tablets and their Combined (relay log) positions are frozen. Today ERS waits for all tablets to apply relay logs before picking a winner — but a tablet whose Combined position is strictly behind the leading group can never win. Waiting for it is pointless and can only hurt us (by timing out the entire ERS over a tablet that was never a contender)

Filtering out known-problem tablets early makes reparents less brittle and faster 🚀

This PR addresses #18529 by making ERS more tolerant of lagging tablets, while preserving the AGENTS.md ERS contract that we must pick the most-advanced candidate with certainty

Note

The split-brain detection / --allow-split-brain-promotion follow-up that originally lived in this PR has moved to #20289 to make review more streamlined

What changed

  1. GTID-only relay-log-apply optimisation — for GTID-based replication, Combined is the received relay-log position, so we can prove which tablets can never win and filter them out of the wait. Tablets at the leading Combined position are waited on; the first success short-circuits the rest. Lagging tablets stay in the candidate pool (they can still be repointed at the new primary later), they just don't block the wait phase. For non-GTID flavours (FilePos, MariaDB) this optimisation is unsafe — Combined there is the executed position and doesn't reflect received-but-not-applied relay logs — so those flavours keep the pre-PR requireAll=true behaviour unchanged

  2. Pairwise-dominance filterfilterToMostAdvancedCombined() uses pairwise dominance (not "compare to a single max") so it correctly handles partially-ordered GTID sets: when two candidates have disjoint UUIDs neither dominates the other, so both stay in the leading group. Comparing against a single chosen max would silently drop one incomparable maximum

  3. Errant-GTID re-wait pass — errant filtering can remove every originally-applied tablet, leaving only "unwaited survivors" (tablets cancelled mid-apply by our own short-circuit, or strictly-behind tablets we excluded from the first wait). Promoting one of those would risk a primary with received-but-unapplied transactions 😱 We detect this case (no applied tablet survives errant detection) and run a second filter → wait pipeline over the survivors before promotion

Implementation details

A few smaller mechanics behind the headline changes:

  • applyRelayLogsAndReconcile() helper — applied tablets get Executed bumped to Combined so the existing sorter prefers them via the CombinedExecuted → promotion-rule tiebreak; failed tablets are removed from validCandidates, cancelled ones are left untouched
  • Pre-satisfied PRIMARY-like candidates — a tablet absent from statusMap (current/stuck PRIMARY from ErrNotReplica) has no relay logs to apply, so it's marked true in successMap upfront. This bumps its Executed = Combined in reconcile, preventing cancelled-mid-apply peers (non-zero pre-wait Executed) from sorting ahead of it in findMostAdvanced
  • Cancellation classification — errors from our own groupCancel() or from a cancelled parent context are treated as expected noise (must check both errors.Is(err, context.Canceled) and gRPC-wrapped vtrpc.Code_CANCELED, since the latter does not satisfy the former). When the parent ctx is cancelled with no real failure to surface, we return the wrapped ctx.Err() directly so operators see "aborted while waiting for relay logs to apply"
  • Deferred-cleanup safeguard — once PromoteReplica/InitPrimary returns nil on the promotion target, the deferred replication-restart cleanup must not run on that tablet (would call StartReplication on a now-PRIMARY). A primaryPromoted flag is plumbed through reparentReplicas so the cleanup filters the promoted tablet out

Stats

Two new stats are exported for observability:

  • EmergencyReparentFilteredCandidates{Keyspace, Shard} — tablets excluded from the relay-log wait because their Combined position is strictly behind the leading group
  • EmergencyReparentRelayLogFailedCandidates{Keyspace, Shard} — tablets that genuinely failed to apply relay logs (RPC error, MySQL error, or timeout). Cancellations after a peer succeeded — or after parent-ctx cancel — are not counted. The metric is also incremented on the error path before any abort returns, so operators can still see failure counts when ERS aborts for some other reason

Testing

E2E tests in ers_test.go:

  • TestERSFiltersNonMostAdvancedCandidates — stops the IO thread on one replica, writes data the lagger won't see, kills the primary, runs ERS, then asserts EmergencyReparentFilteredCandidates incremented via the vtctld /debug/vars endpoint and that the lagging tablet was not chosen as the new primary
  • TestReplicationStopped updated — previously asserted ERS failed with 2 replicas having replication stopped; now asserts ERS succeeds with the third tablet as the surviving candidate, since partial relay-log failures are tolerated
  • Both new behavioural tests are gated on e2eutils.SkipIfBinaryIsBelowVersion(t, 25, "vtctld") so the Reparent Old Vtctl upgrade-downgrade job (which runs against the previous release's vtctld) skips them cleanly

Related Issue(s)

Resolves: #18529

Checklist

  • "Backport to:" labels have been added if this change should be back-ported to release branches
  • If this change is to be back-ported to previous releases, a justification is included in the PR description
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on CI?
  • Documentation was added or is not required

Deployment Notes

This is a user-visible behavioural change in EmergencyReparentShard and is called out in changelog/25.0/25.0.0/summary.md under VTCtld. No new required flags or configuration:

  1. GTID-based shards — ERS now tolerates a minority of replicas failing or hanging during the relay-log-apply wait. As long as at least one tablet at the leading Combined position applies successfully (or a no-status PRIMARY-like tablet is present at that position), ERS proceeds. The pre-PR behaviour (any single failure aborts ERS) was unnecessarily fragile

  2. Non-GTID shards (FilePos, MariaDB) — behaviour is unchanged. The optimisation is gated on isGTIDBased and these flavours still wait for every candidate

Two new stats (EmergencyReparentFilteredCandidates, EmergencyReparentRelayLogFailedCandidates) are exported from vtctld for operators to track filter and failure counts per keyspace/shard

AI Disclosure

Claude Code assisted with development, testing and this PR summary

@vitess-bot

vitess-bot Bot commented Sep 30, 2025

Copy link
Copy Markdown
Contributor

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

  • Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

  • Apply the release notes (needs details) label if users need to know about this change.
  • New features should be documented.
  • There should be some code comments as to why things are implemented the way they are.
  • There should be a comment at the top of each new or modified test to explain what the test does.

New flags

  • Is this flag really necessary?
  • Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
  • vtctl command output order should be stable and awk-able.

@vitess-bot vitess-bot Bot added NeedsBackportReason If backport labels have been applied to a PR, a justification is required NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels Sep 30, 2025
@github-actions github-actions Bot added this to the v23.0.0 milestone Sep 30, 2025
@timvaillancourt timvaillancourt added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: VTOrc Vitess Orchestrator integration Component: vtctl and removed NeedsWebsiteDocsUpdate What it says NeedsBackportReason If backport labels have been applied to a PR, a justification is required NeedsIssue A linked issue is missing for this Pull Request labels Sep 30, 2025
@codecov

codecov Bot commented Sep 30, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.90909% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.17%. Comparing base (70c7a72) to head (8f9fbdf).
⚠️ Report is 319 commits behind head on main.

Files with missing lines Patch % Lines
go/vt/vtctl/reparentutil/emergency_reparenter.go 89.32% 11 Missing ⚠️
go/vt/vtctl/grpcvtctldserver/server.go 66.66% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main   #18707       +/-   ##
===========================================
+ Coverage   69.67%   76.17%    +6.50%     
===========================================
  Files        1614       18     -1596     
  Lines      216793     6229   -210564     
===========================================
- Hits       151044     4745   -146299     
+ Misses      65749     1484    -64265     
Flag Coverage Δ
partial 76.17% <90.90%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@timvaillancourt timvaillancourt removed the NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work label Sep 30, 2025
@timvaillancourt timvaillancourt marked this pull request as ready for review September 30, 2025 14:50
@timvaillancourt

timvaillancourt commented Sep 30, 2025

Copy link
Copy Markdown
Contributor Author

Copying some conclusions from an offline discussion with @arthurschreiber:

  • As it stands this PR will avoid a minority of lagging tablets from preventing ERS ✅
  • As this PR stands, ERS will still-fail if a single or "majority" is picked and any of those tablets fail to apply logs, because the code still expects 100% of candidates to succeed 🟡
  • It would be ideal if ERS tried the "next-best" candidate. Today it does not 🟡
    • This means selecting a 1+/majority may still be beneficial, if we know how to handle partial results
    • Call-out: in some case picking a next-best candidate will result in an errant GTID. Today the code makes every effort to avoid this, to the point of erring on failing the ERS. This is good for correctness but bad for availability. Some users may have differing views on this tradeoff. This should probably be configurable

@mattlord

mattlord commented Oct 1, 2025

Copy link
Copy Markdown
Member

The reason, AFAIUI, for the current behavior is to prevent any of the healthy tablets from becoming forever unhealthy/unusable due to the new primary not having binary logs covering/containing the GTIDs that the lagging replica(s) may still need (the new primary may have recently been restored from a backup and have minimal binary logs). Have you already thought about that in this context?

@timvaillancourt

timvaillancourt commented Oct 1, 2025

Copy link
Copy Markdown
Contributor Author

The reason, AFAIUI, for the current behavior is to prevent any of the healthy tablets from becoming forever unhealthy/unusable due to the new primary not having binary logs covering/containing the GTIDs that the lagging replica(s) may still need (the new primary may have recently been restored from a backup and have minimal binary logs). Have you already thought about that in this context?

@mattlord I don't think that has changed, but I would appreciate you double checking my assumption because that is a very important functionality

One part of the code that could affect that was actually changed in #18531. Previous to this PR, the code called position.AtLeast(otherPos) on replication.Positions and this PR moved things to call a wrapper (*reparentutil.RelayLogPositions) with the same method name (.AtLeast(...)) that calls 2 x different replication.Positions: https://github.com/timvaillancourt/vitess/blob/main/go/vt/vtctl/reparentutil/replication.go#L55-L69

The TL;DR on that wrapper func: we do the same sort but put now prioritise positions with the most advanced SQL thread if two combined sets are equal. In the end the same replication.Position is called unchanged and that is what is deciding which GTID set is larger

And in terms of being certain we're ignoring the right tablets: after StopReplicationAndGetStatus is ran replication is stopped, we know the GTID sets and replication is not started until after the post-wait-for-relaylogs candidate selection. That selection uses the same sort logic as this optimization. So, the idea is: because replication isn't moving and we know all GTIDs each candidate could potentially apply when asked, we already know the post-wait-for-relaylogs GTID sets each candidate could have. This means they can be filtered before the apply phase, instead of after. Or TL;DR: we already know the losers after running StopReplicationAndGetStatus RPCs (via the After-field GTIDSets)

@mattlord mattlord self-assigned this Oct 2, 2025
@systay systay modified the milestones: v23.0.0, v24.0.0 Oct 8, 2025
@timvaillancourt timvaillancourt marked this pull request as draft November 3, 2025 19:36
@github-actions github-actions Bot added the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Dec 4, 2025
@timvaillancourt timvaillancourt removed the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Dec 4, 2025
@github-actions github-actions Bot added the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Jan 4, 2026
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
@timvaillancourt

Copy link
Copy Markdown
Contributor Author

@mattlord thanks for the review!

I'm leaning towards keeping it together, but let me know if separating the "split-brain" scenario makes this an easier review. Shipping together makes it more clear it will all work together though

We should not let split-brain override promote a lagging survivor in go/vt/vtctl/reparentutil/emergency_reparenter.go:331-342. With AllowSplitBrainPromotion=true, errant-GTID detection restores the pre-detection set only when it removes every candidate. In the mixed case of two mutually-errant leading candidates plus a lagging survivor, both leaders can be pruned while the lagger remains, so ERS can re-wait and promote the lagger. That loses the unique writes from both split-brain sides, which is worse than the advertised “pick one side; losing side becomes errant” semantics. Please either require/validate --new-primary for this override path or restore/limit to the pre-errant leading set when all leading split-brain candidates are pruned and add a regression test with two mutually-errant leaders plus a lagging survivor.

Good idea, implemented in ef09389

We should preserve SQL-thread state during failed-ERS cleanup in go/vt/vtctl/reparentutil/replication.go:283-285, go/vt/vtctl/reparentutil/emergency_reparenter.go:487-490. The new cleanup restarts any replica whose IO thread was healthy before ERS stopped it, but it calls StartReplication, which starts both IO and SQL threads. If a replica intentionally had SQL_THREAD stopped but IO running before ERS, an early abort will restart SQL unexpectedly. Cleanup should restore the original thread state exactly, or avoid calling full StartReplication for SQL-stopped replicas and cover that case with a unit test.

Sounds good 👍. Implemented in ef09389

…-lagging

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

This comment was marked as outdated.

@mattlord mattlord left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In go/vt/vtctl/reparentutil/emergency_reparenter.go:438-440 we shouldn't drop failed-ERS replication cleanup before reparenting has actually recovered the replicas. At this point ERS has only applied relay logs; replicas whose IO_THREAD was stopped are still stopped. Clearing replicasToRestart here means later aborts, such as promoteIntermediateSource failure, waitForCatchUp timeout, lost topo lock, or final promotion/repoint failure, skips deferred cleanup and can leave replicas with IO stopped. Please keep cleanup state until affected tablets have actually been repointed/started, or track per-tablet state, and add a regression test that fails after this point.

The text in changelog/25.0/25.0.0/summary.md:140-143 around the split-brain override limitations are stale. The code now restores the leading set when no leading candidate survives errant-GTID detection, so the “lagging tablet can still be picked” warning is no longer accurate. The sorter also now has deterministic promotion-rule/alias tie-breakers, so the map-iteration warning is misleading. Please update the release note to match the current behavior. Actually... I would just remove this from the summary. This is closer to a bug fix and doesn't need to be highlighted in the v25 release summary IMO.

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
@timvaillancourt

Copy link
Copy Markdown
Contributor Author

In go/vt/vtctl/reparentutil/emergency_reparenter.go:438-440 we shouldn't drop failed-ERS replication cleanup before reparenting has actually recovered the replicas. At this point ERS has only applied relay logs; replicas whose IO_THREAD was stopped are still stopped. Clearing replicasToRestart here means later aborts, such as promoteIntermediateSource failure, waitForCatchUp timeout, lost topo lock, or final promotion/repoint failure, skips deferred cleanup and can leave replicas with IO stopped. Please keep cleanup state until affected tablets have actually been repointed/started, or track per-tablet state, and add a regression test that fails after this point.

The text in changelog/25.0/25.0.0/summary.md:140-143 around the split-brain override limitations are stale. The code now restores the leading set when no leading candidate survives errant-GTID detection, so the “lagging tablet can still be picked” warning is no longer accurate. The sorter also now has deterministic promotion-rule/alias tie-breakers, so the map-iteration warning is misleading. Please update the release note to match the current behavior. Actually... I would just remove this from the summary. This is closer to a bug fix and doesn't need to be highlighted in the v25 release summary IMO.

@mattlord good points, updated in a7262cb 👍

Comment thread go/vt/vtctl/reparentutil/emergency_reparenter.go
Comment thread CLAUDE.md Outdated

@mattlord mattlord left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In go/vt/vtctl/reparentutil/emergency_reparenter.go:438-483 the cleanup list now survives into the final promotion/reparent phase, so I think that any error after PromoteReplica/InitPrimary succeeds can trigger the deferred cleanup and call StartReplication on the just-promoted primary. That can then run START REPLICA on a primary, apply replica semisync settings to it, and wrap/mask the real ERS failure. The new regression test covers PromoteReplica failing, but not failures after promotion succeeds, e.g. PopulateReparentJournal or later replica-repointing errors. I think that we should keep cleanup for pre-promotion aborts, but remove/prune the primary-elect from replicasToRestart once promotion succeeds, and add a post-promotion failure regression test.

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

This comment was marked as outdated.

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
@timvaillancourt

timvaillancourt commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

@arthurschreiber / @mattlord thanks for reviews!

I believe I have address the open concerns, please validate when you have time 🙇

@mattlord mattlord left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In go/vt/vtctl/reparentutil/emergency_reparenter.go:202-245, 516-540 the deferred failed-ERS cleanup still runs after PromoteReplica/InitPrimary succeeds and only filters out the promoted tablet. If PopulateReparentJournal or the final SetReplicationSource fanout fails after the new primary is already writable, cleanup calls StartReplication on the other tablets whose IO thread ERS stopped. StartReplication does not repoint them; it resumes their existing source and applies semisync relative to prevPrimary, which can reconnect replicas to the old primary/source after a new primary has been promoted. Please disable this generic pre-promotion cleanup once promotion succeeds, or track per-tablet final-repoint success and only perform a deliberate post-promotion recovery against the new primary.

In go/vt/vtctl/reparentutil/replication.go:286-303 the SQL-stopped cleanup path still does not restore the pre-ERS state. For tablets with IO running and SQL intentionally stopped, ERS stops IO via StopReplicationAndGetStatus(... IOTHREADONLY), but replicasWithStoppedIO now excludes them from cleanup to avoid StartReplication starting SQL. That avoids starting SQL, but leaves IO stopped after a pre-promotion abort. Please restore the exact original thread state, e.g. start only IO for this class, and add a regression that verifies SQL remains stopped while IO is restarted.

@timvaillancourt

timvaillancourt commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the careful re-read, @mattlord. Splitting these:

Issue 1 (post-promotion cleanup): Agreed, will fix in the next push. Approach: skip the deferred StartReplication cleanup entirely once primaryPromoted == true. The prevPrimary source on the remaining replicas is stale after promotion, and StartReplication doesn't repoint — it just resumes the existing source, which would reconnect those tablets to the old/dead primary while a new primary is already writable. Disabling the generic cleanup at that point is simpler than per-tablet repoint tracking + an RPC against the new primary, and matches the AGENTS.md guidance to reduce points of failure in the abort path. I'll add a regression test where PromoteReplica succeeds and PopulateReparentJournal fails, asserting cleanup runs zero times.

Issue 2 (SQL-stopped class): Valid improvement, but I'd like to defer it to a follow-up for scope/review velocity if you're OK with that.

Quick context: in your earlier review on this PR (#18707 (review)) you offered two options for the SQL-stopped class — "restore the original thread state exactly, or avoid calling full StartReplication for SQL-stopped replicas and cover that case with a unit test." This PR took the second one: the skippedSQLStopped split in replicasWithStoppedIO plus the warning log plus the TestReplicasWithStoppedIO unit test. So the silent SQL re-start you flagged is fixed today.

The exact-state restoration you're now describing (start IO, leave SQL stopped) is the stronger fix, but it needs a new tabletmanager capability — no current RPC issues START REPLICA IO_THREAD. Adding one touches proto + generated code + tabletmanager + tmclient + grpc client/server + mysqlctl + flavor commands + fakes/mocks, plus an e2e for the IO-running/SQL-stopped path. I'd rather land that as a focused follow-up where it can get its own design discussion, instead of bolting it onto this PR.

If you'd rather have it in this PR I'll do it — just wanted to flag the scope first.

- skip failed-ERS cleanup once promotion has been attempted (mattlord);
  flip primaryPromoted before InitPrimary/PromoteReplica so a post-side-
  effect RPC error doesn't run StartReplication against a tablet that may
  already be writable (codex P1)
- findMostAdvanced Safety net #2: reject applied candidates strictly
  dominated by another eligible applied candidate (codex P1)

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

This comment was marked as outdated.

…agging

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

# Conflicts:
#	changelog/25.0/25.0.0/summary.md
Comment thread CLAUDE.md
@timvaillancourt

timvaillancourt commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Update: converting to draft as I separate the GTID-based optimization and the split-brain fix, as discussed, in order to make review more streamlined

Strip the split-brain-detection feature from this branch so reviewers can
evaluate the GTID-based ERS optimization on its own. The split-brain pieces
(`AllowSplitBrainPromotion` flag, upfront uniformity check, errant-GTID
restoration safeguard, override paths in findMostAdvanced) move to a
separate ers-split-brain-detection branch built independently on main.

What stays here:
- Filter to the leading Combined position before relay-log apply, tolerate
  partial relay-log-apply failures, and bump applied tablets' Executed to
  Combined so the sorter prefers them.
- replicasWithStoppedIO returns SQL-stopped replicas as a separate slice so
  cleanup can skip-and-warn instead of silently restarting them.
- reparent_sorter handles incomparable GTID positions deterministically.
- newPrimary/primaryPromoted deferred-cleanup safeguard.
- EmergencyReparentFilteredCandidates and EmergencyReparentRelayLogFailedCandidates stats.

AGENTS.md and the changelog still describe both features and reach their
full state once the split-brain follow-up also lands.

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

This comment was marked as outdated.

The split-brain detection (upfront FAILED_PRECONDITION abort),
--allow-split-brain-promotion flag, and EmergencyReparentSplitBrainOverrides
stat are not part of this branch's code. They moved to the follow-up
ers-split-brain-detection PR. This commit strips the changelog text that
described them so the release notes match what's actually delivered here.

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

This comment was marked as outdated.

…toff

Five fixes from successive codex and Claude Opus paranoid reviews:

1. First-wait guard: when filterToLeadingGroup keeps multiple non-dominated
   candidates (incomparable Combined positions), force requireAll=true. The
   one-success short-circuit would otherwise let applyRelayLogsAndReconcile
   delete a failed incomparable leader from validCandidates, bypassing the
   AtLeast split-brain check in findMostAdvanced. Includes a regression test
   verified to fail without the guard.

2. Second-wait guard: same uniformCombined check on the errant-GTID re-wait
   pass, so survivors with incomparable Combined positions are subject to
   the same requireAll-on-failure semantics.

3. Cleanup cutoff: replicationMutated (renamed from primaryPromoted) now
   flips before promoteIntermediateSource — once SetReplicationSource has
   started mutating replicas, the deferred StartReplication cleanup is
   unsafe whether or not we reach the final PromoteReplica.

4. Deadline-as-cancellation: waitForAllRelayLogsToApply now treats every
   waiter error after parent ctx.Err()!=nil as expected noise, not just
   context.Canceled. Parent-ctx DEADLINE_EXCEEDED was previously recorded
   as a per-tablet failure and could bump EmergencyReparentRelayLogFailedCandidates
   on operator timeout.

5. validCandidates non-mutation regression test: asserts that on the
   requireAll-with-failure path, applyRelayLogsAndReconcile returns the
   error before the reconcile loop runs, leaving validCandidates and its
   pointed-to RelayLogPositions bit-identical to the input. Guards (1)
   and (2) depend on this property to be load-bearing.

Reverts the in-PR attempt to leave SQL-thread-stopped replicas at their
pre-ERS state — deferred to vitessio#20256 which adds the missing
StartReplicationMode_IOTHREADONLY primitive across the tabletmanager
surface.

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
…agging

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 11 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component: VTAdmin VTadmin interface Component: vtctl Component: VTOrc Vitess Orchestrator integration Type: Enhancement Logical improvement (somewhere between a bug and feature)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug Report/RFC: lagging tablet(s) can cause EmergencyReparentShard to fail

5 participants